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Abstract 

To encourage the use of research-based information (RBI) in education in Quebec, the Centre de transfert pour la 
reussite educative du Quebec CTREQ and the RENARD team, a knowledge transfer research team, developed the 
Competency Model for Knowledge Translation to Support Educational Achievement among Quebec Youth. They 
then developed a pilot project involving 24 school practitioners working in seven schools in Quebec to validate the 
content of this framework and to encourage its implementation. The aim of the present study was to evaluate the 
processes and outcomes of this pilot project’s implementation. Qualitative methods were used, involving data from: 
1) three series of interviews with project participants, and 2) tracking sheets used to monitor activities carried out by 
project participants as well as by the project leaders supporting them in its implementation. Results showed the RAC 
implementation was able to influence participants’ perceptions regarding their role in the process of fostering RBI 
use and provided them with resources to fulfil that role. It also appeared that participants were able, to varying 
degrees, to extend this learning into their professional practice. The present article also highlights elements of the 
pilot project that facilitated or impeded the achievement of these outcomes and suggests areas to be considered for 
future interventions and studies related to the implementation of a competency model to encourage RBI use. 

Keywords: Knowledge transfer, Competency model. Theory-driven evaluation 
1. Theoretical Framework 

1.1 Review of Literature 

In education, as in other fields such as management, health, and psychosocial services, there is still a profound gap 
between the production of research-based information (RBI) and its use, despite growing interest and increased 
efforts to encourage such knowledge transfer (Dagenais et al., 2010, 2012; Berube, 2006; Bowen & Graham, 2013; 
Davies, Nutley, & Walter, 2008; Graham & Tetroe, 2007; Graham et al., 2006; Hemsley-Brown & Shaip, 2003; 
Landry et al., 2008; Lysenko et al., 2014; Organisation for Economic Co-operation and Development, 2000, 2007, 
2010; Rohrbach et al., 2005; Walter et al., 2004; Williams & Coles, 2007). A survey conducted by Quebec’s Council 
for Higher Education (2006) found that, even though school practitioners in Quebec’s education sector demonstrated 
an openness to RBI, they were not inclined to incorporate it into their practice. The Council, whose mandate is to 
advise the Ministry of Education, Leisure and Sports, concluded its report by stressing that more must be done to 
develop a culture of research and innovation that will encourage RBI use among practitioners in the education sector. 
It was with this in mind that the Centre de transfert pour la reussite educative du Quebec (CTREQ), whose mission 
is to “promote innovation and knowledge transfer to increase educational success in Quebec” 
(http://www.ctreq.qc.ca), implemented a project in collaboration with the education sector to identify and develop 
school practitioners’ competencies to promote RBI use in their schools. In this endeavour, the CTREQ worked with 
the RENARD team, a knowledge transfer research team (http:// www.equiperenard.ca ), to develop the Competency 
Model for Knowledge Translation to Support Educational Achievement among Quebec Youth (RAC - Referentiel 
d’agir competent a l'integration de connaissances favorables a la reussite educative des jeunes du Quebec). 

A competency model is a normative and prescriptive document that provides a modelling of competencies related to 
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a given activity or role and specifies the conditions and methods for implementing these competencies, as well as the 
conditions and criteria for evaluating them (Cherqui-Houot, 2009). The main objective of a competency model is 
usually to strategically influence the behaviours of a given public (usually the members of an organization) by setting 
out the behavioural themes or types of behaviours that are expected and rewarded (Redmond, 2013). For such a 
document to be well received by the targeted public and its implementation to succeed, careful planning is required 
(Soderquist, Papalexandris, Ioannou, & Prastacos, 2010). In essence, operationalizing the organizational change that 
is sought by implementing such a model is complex and the success of such an undertaking cannot depend solely on 
the document’s content or formulation (Soderquist et al., 2010). That being said, few studies have examined how the 
individuals targeted by a model perceive it and react to its implementation (Caldwell, 2008; Heinsman, de Hoog, 
Koopman, & van Muijen, 2008; Redmond, 2013). 

For this reason, the CTREQ and the RENARD team set out to evaluation the outcomes and processes of the RAC 
implementation pilot project. The evaluation was inspired by Chen’s (2015) theory-driven integrative approach for 
evaluating processes and outcomes. It is presented in three stages: 1) clarifying the program theory; 2) collecting and 
analyzing the relevant data with respect to the program theory; and 3) critically assessing the program theory 
components and the links between them. 

1.2 Clarifying the Program Theory 

In the first step, the evaluation team worked with the three members of the scientific committee—project leaders, in 
charge of developing and implementing the project—to establish the program theory. This theory had two 
components: the change model and the action model on which the program was based. The change model presented 
the descriptive assumptions of the program, i.e., everything having to do with the causal processes underpinning the 
program. More concretely, a change model is used to establish clearly the program’s objectives and the determinants 
upon which it is intended to act to achieve those objectives (Chen, 2015). The action model, on the other hand, 
described the prescriptive assumptions of the program. In other words, it presented the components and activities that 
were prescribed by the program in order to act on the determinants and achieve the objectives described in the 
change model (Chen, 2015). The program theory produced in this first step will be described here only briefly; for an 
exhaustive description, the reader is referred to Briand-Lamarche, Pinard, Theriault & Dagenais (in preparation). 

1.2.1 The Change Model 

The change model for the RAC implementation pilot project shows that the aim of the project was to increase RBI 
used by members of the staff of the participating schools to improve their teaching practices and thereby enhance 
their students’ learning. To this end, the RAC implementation was focused on improving the individual and 
collective competencies for RBI use of a triad of actors (a teacher, an educational advisor, and the school principal) 
from each of the seven participating schools. To achieve its objectives, the project targeted three determinants: 1) 
influencing participants’ perceptions regarding their role in the process of fostering RBI use; 2) increasing 
participants’ resources with regard to RBI use, i.e., giving them tools to help them better fulfil their role; and finally, 
3) supporting the extension of participants’ learning into their schools—that is, enabling them to use the perceptions 
and tools they had developed so they could function more competently in implementing a process to support RBI use 
in their school. 

In view of the literature, these aims appear plausible. Pertaining to the first determinant, previous research stated that 
members of an organization will perform better when they have a better understanding of their role and of the 
organization’s expectations regarding those roles (Becker & Huselid, 1999; Campion et al., 2011; Suter, Arndt, 
Arthur, Parboosingh, Taylor, & Deutschlander, 2009). Furthermore, the implementation of competency model has 
been proven helpful to clarify professional roles (Reeves, Fox, & Hodges, 2009). As for the other two determinant, a 
recent systematic review regarding what works in knowledge transfer established that developing skills in accessing 
and making sense of RBI is an effective way to promote RBI utilization (Breckon and Dodson, 2016). 

1.2.2 The Action Model 

The action model for the RAC implementation project presents the components and activities intended to produce 
the determinants (i.e., the action mechanisms) and the objectives described in the change model presented above. 
The action model includes six components: the implementing organizations; the implementers; the associated 
organizations; the intervention and service delivery protocols; the target population; and finally, its context. The 
model indicates that two organizations were responsible for the project and five organizations or committees were 
associated with the RAC implementation project. 

With regard to the context within which the RAC implementation project was conducted, two points must be 
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considered: 1) the micro context, which presents the elements connected with social, psychological, and material 
support that can influence participation in the program; and 2) the macro context, which includes the social, cultural, 
political, and economic standards that can influence the program’s implementation. In terms of the micro context, 
several elements were incorporated into the RAC implementation project to support participation, the main one being 
the refund of various costs associated with participation, in order to support the schools in implementing the project. 
Regarding the macro context, social and cultural standards in the education sector appear increasingly to value RBI 
use, thereby presenting a supportive context for the project’s implementation. 

The intervention itself includes nine components, of which the most salient are: 1) the RAC itself (including a 
document putting the project in context, and nine cards presenting the targeted competencies and the resources 
needed); 2) a one-and-a-half-day training session to present the RAC to participants and discuss various topics 
related to the process of promoting RBI use in education; and 3) four community of practice meetings over a 
one-year period. 

The target population and the participation process, i.e., the process followed by the participating triads in joining the 
project, involved two steps: selecting the schools and selecting the members of each school’s triad. At both these 
levels, participation had to be voluntary, and the primary selection criterion was being open to using RBI. That being 
said, a large portion of this process was variable, as it was left to the school settings to select the participants 
(educational advisor, teacher, school principals), in order to avoid any meddling with internal management 
processes. 

1.3 Objectives of this Study 

After the program theory was developed in collaboration with the project’s scientific committee, data were collected 
and analyzed to critically review the components of this theory and the links between them, as recommended in 
Chen’s (2015) theory-driven process and outcomes evaluation approach. The results of that evaluation, presented in 
this article, address two questions: 1) What were the proximal outcomes of the RAC implementation, i.e., what 
determinants were activated by this implementation? 2) Which components of the action theory facilitated or 
impeded activation of the determinants? 

2. Methodology 

To critically assess the program theory of the RAC implementation pilot project, the evaluation was conducted using 
a multiple case study in which each school represented one case. Multiple case analysis is used to highlight elements 
that are similar and different among cases (i.e., from one participating school to another) to gain a clear and in-depth 
understanding of the intervention and its outcomes (Stake, 2006). A summary of the methodological process is 
presented in Table 1. 

2.1 Participant Selection and Data Collection Procedures 

A total of seven schools were involved in this project. In each of these schools, a triad of actors was selected to 
participate: an educational advisor, a teacher and the school principal. One school decided to include a second 
educational advisor and two schools added a second teacher, for a total of 24 project participants. In addition, three 
school principals were replaced in the summer of 2014, which was before the start of the project’s implementation in 
the schools but after the training to prepare for the implementation, conducted in May 2014. One teacher abandoned 
the project during the year. Two of the schools were high schools (one urban and the other semi-rural) and five were 
primary schools (one urban, three semi-rural, and one rural). All members of the project triads (24 individuals) were 
invited to participate in each step of the evaluation (three series of interviews and completion of tracking sheets). The 
following section presents, in greater detail, the characteristics of these evaluation steps and of the participating triad 
members. The first method used to collect data was a series of semi-structured interviews conducted in June 2014, a 
few weeks after the training. The aim of these interviews was to document participants’ reactions to the training; how 
they envisioned the application, in their schools, of the knowledge acquired from the training and the type of support 
they thought they would need to help them use the knowledge in this way. Participants who had indicated an interest 
in being interviewed were contacted by email. Of the 24 individuals who went through the training, 12 indicated they 
were willing to be interviewed. However, when those 12 were contacted by email for this purpose, only seven made 
appointments to be interviewed. Of these, six were educational advisors and one was a school administrator. They 
were from five of the seven participating schools (one high school and four primary schools). Interviews were 
conducted by telephone by the first author of this article. At the start of the interview, participants were advised that 
the content of the interview was confidential and that, in any communications or publications relating to the results 
of this evaluation, it would not be possible to identify them specifically. Participants were also advised that they 
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could stop the interview at any time, without consequence, and that, with their consent, the interview would be 
audio-recorded to facilitate data analysis. 

2.1.1 Interview Series on the Implementation and Outcomes 

Two other series of semi-structured interviews were conducted, in November 2014 and June/July 2015. The purpose 
of those interviews was to document participants’ reactions to the activities implemented as part of the intervention, 
as well as to document the impacts of these activities on: a) their perceptions of their role in RBI use; b) the level to 
which they were equipped to carry out this role; and c) the degree to which they had extended their knowledge into 
their practice. The procedures used to conduct both series of interviews were the same. First, all project participants 
(n = 24) were contacted by email to explain that telephone interviews would be conducted in the coming weeks 1 , to 
present the objective of the interview and its expected duration (about one hour), and to invite those who were 
interested to reply with a list of available dates and times so that a telephone appointment could be confirmed. Two 
weeks after the first mailing, a reminder was sent, again by email. Lastly, information to participants regarding 
confidentiality, audio-recording of the interview, and the option of stopping the interview were the same as in the 
first series of interviews described above. In the November 2014 interview series, 20 out of 24 participants in the 
RAC implementation project accepted to be interviewed, and in the June-July 2015 series, 17 out of 24 participants 
responded. Of the 24 project participants, only one was not interviewed in either of these two exercises. 

2.1.2 Tracking Sheets 

Tracking sheets were used to document participants’ implementation of the project in their schools. Two types of 
sheets were developed for this purpose: one for the project participants and another for project leaders from the 
scientific committee. Participants completed their tracking sheet each time they implemented an activity or 
intervention to promote RBI use in their schools. This sheet contained 12 questions used to identify the person 
completing the sheet, describe the activity or intervention conducted in their school, and target the competencies and 
resources required to conduct the activity or intervention. The second tracking sheet was to be completed by project 
leaders each time they offered individual support to a participant. This sheet consisted of 11 questions to document 
every action carried out to provide individual support. In all, 122 sheets were completed by project participants. Of 
these, 78 were completed by educational advisors, 10 by school principals, and one by a teacher. Twenty were 
completed by two members of a same triad together, and the remaining 13 were completed in cooperation by all the 
members of a given triad. The number of sheets varied considerably from one school to another, ranging from 4 to 
35 over the pilot project year. The project leaders completed 25 tracking sheets. 

2.2 Data Analysis 

All of the data collected were entered into a computer-assisted qualitative analysis program (NVivo 10). Interviews 
were recorded with the participants’ consent and the digital files were imported into the software. The interview 
content was partially transcribed and a feature of the software kept each segment of the transcription linked to its 
audio track segment. With this feature, it is possible, with a single click, to re-listen to the various audio segments 
linked to the transcript, in order to be able to take into account, in the analysis, the exact formulation of the 
participant’s statement, as well as his or her intonation, hesitations, and silences. 

Once transcribed, the interview contents were analyzed using the method proposed by Miles and Huberman (2003). 
This method involved three phases. First, the data were condensed, i.e., they were broken down according to codes 
corresponding to the themes addressed (e.g. collaboration, roles, resources). These codes, which could be applied to 
a word, a sentence, or an entire paragraph, were used to organize participants’ statements based on the ideas and 
themes contained, so that analytical operations could then be carried out (Lessard-lTebert et al., 1996). A coding grid 
was developed prior to the analysis based on the broad themes covered by the interview questions, while leaving 
sufficient room for new codes emerging from participants’ statements. 

In a second phase, the data were organized. In this phase, analytical operations were carried out based on the codes 
applied in the preceding phase, so that participants’ statements could be organized for interpretation. For this purpose, 
a summary sheet was completed for each school involved in the project. These sheets collected in one location, for 
each school, the key information obtained in the interviews on each theme. 

The final phase of the analysis consisted of interpreting the data. This involved producing knowledge based on the 
organized statements (Miles & Fluberman, 2003). Flowever, before proceeding with this final phase, the data from 
the tracking sheets completed by the participants and the project leaders were also analyzed using the method 
described in the two previous phases. This allowed the contents of these tracking sheets to be added to the summaries 
for each school. Finally, in this third and final phase, which was data interpretation, the seven summary sheets (one 
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for each school) were compared to identify similarities and differences in content from one school to another. To do 
this comparison, an analysis sheet was created for each component of the action model as well as for each 
determinant in the change model. All of the content of the seven summaries was then re-read and incorporated into 
the analysis sheets for the relevant components or determinants. In this way, this phase enabled the links to be 
documented between the different components of the action model and the deployment of the determinants identified 
in the change model, while keeping in mind the nuances related to each school and profession. 


Table 1. Summary of methodological process 
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3. Results 


This section presents the results revealed by the data that were collected and analyzed in relation to the components 
of the program theory and the links between these components. In this section, two questions are addressed: 1) What 
were the proximal outcomes of the RAC implementation, i.e., what determinants were activated by this 
implementation? 2) Which components of the action theory facilitated or impeded activation of the determinants? 

3.1 What Determinants Were Activated by the RAC Implementation? 

To answer this question, the evaluation sought to determine whether the RAC implementation project enabled 
activation of the three mechanisms identified as determinants in the program’s change model: 1) influencing 
participants’ perceptions regarding their role in the process of fostering RBI use; 2) increasing participants’ resources 
with regard to RBI use; and 3) supporting the extension of learning to other practitioners in the participating schools. 

3.1.1 Influencing Participants’ Perceptions regarding Their Role in the Process of Fostering RBI Use 

Results indicate that the project was able to influence participants’ perceptions of their role in the process of 
fostering RBI use. Even though they already conceived role quite similarly to what was proposed by the RAC, the 
project allowed them to re-examine some of their practices. 

It was clear, blit now it’s better defined. That is, in action, we see our role, we know it; we knew it 
beforehand, but now you really went into the details... so now we can say: ‘oh, yes, that I do easily, that I 
do a lot, that I probably don’t do often enough, and, oh, that—do I really do it in my practice? ’ For me, this 
really helped me in my reflexive practice. -Educational advisor (School 6) 

Some differences could be seen at this level among the different professional groups targeted (teachers, school 
principals, educational advisors). 

Teachers. Several teachers reported that the RAC implementation helped them to be more aware not only of the 
need to become informed so they could keep their practices up-to-date, but also, and especially, of the need to use 
reliable sources: 

I’m more critical when someone tells me something relating to research. I’ll delve into it a little further. I 
question, I check things. The training really had that effect: I look into things a bit more. Sometimes things 
happen to us and we take them at face value because that’s what we were given, that’s it, but not 
necessarily. -Teacher (School 4) 

Some also pointed out the fact that, over the course of this project, they had either realized or reinforced their 
perception that they were innovative teachers, and that they had an expertise to share with their colleagues. In this 
respect, they said they were more aware of the unique contribution they could bring to their colleagues and were 
more motivated to share with them, even though some were not always completely comfortable doing so. Several 
teachers also stated that they had developed, over the course of the project, a new or growing interest in evaluating 
the impacts on their students of their changes in practice and that they had become more attentive to these impacts. 

Educational advisors. The educational advisors explained that, overall, the project’s implementation had made 
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them more conscious of the importance of checking regularly to keep up-to-date and not just when an urgent 
situation arose requiring a response to a specific need. Several also indicated that when selecting, adapting, and 
sharing RBI, they were now more sensitive to the needs and characteristics of the teachers with whom they worked. 

School principals. Several school principals indicated that they had discovered, in the community of practice 
meetings, the pleasures and value of learning about other schools’ practices and creating connections. This appeared 
to have inspired some of them to become more networked, to ask questions, to share their vision, and to seek out 
others’ ideas to fuel their own professional development and that of their school team. 

The whole value of networking, that’s something we don’t pay much attention to.... I found that really 
interesting and I told myself that I would like to try’ it, sometimes, when I have a problem... because when 
we stay in our own environment, when it’s our own colleagues, it’s always the same vision... the importance 
of the network means something to me, and I wonder how I might start that up, to help me in my own 
development, because often we’re left to develop on our own, whereas that’s a good way to learn and 
grow... -School principal (School 7) 

In addition, some school principals, as well as certain educational advisors, said the project implementation showed 
them they could be more proactive in their schools. They realized that they should make an effort to find out more 
about teachers’ needs with respect to new knowledge. They also discovered, from this project, that they could call on 
researchers when seeking pedagogical solutions or answers. 

3.1.2 Increasing Participants’ Resources with regard to RBI Use 
In terms of increasing resources, results were more mixed. 

Teachers. From the teachers’ standpoint, it seemed the RAC was not specific enough to be a resource that 
would guide action. In essence, according to the participants, the teacher’s role as defined in the model was very 
much that of any teacher in the school and not the specific role of the teachers involved in the project. Because of 
this, participants said they had difficulty identifying, in the model, any actions pertaining to the role they were 
supposed to play, not only in the triad, but also with their colleagues in the school: 

Basically, the teacher in this project, if he plays his role as a teacher, isn ’t inclined to have a vision for his 
colleagues; he works for himself. If the title of the card was Teacher-Navigator or Teacher-Mentor, maybe 
that would work.... it’s confusing: do I have a responsibility vis-a-vis my colleagues? Should I be pushing a 
bit? Opening some doors? There would be something to do on this card here, but fundamentally, it depends 
what you want to develop: [do you want] mentors who will carry’ this, or do you want to see that whatever 
the teacher in the triad does is then done by all the teachers? -Educational advisor (School 3) 

Moreover, several teachers reported that, while they had developed, over the course of the project, a greater 
awareness of their role in evaluating the impacts of their changes in practice, they had not been given the tools and 
resources needed to carry out this evaluation. 

Educational advisors. The RAC and the tools developed during the project provided certain actions and 
resources that the educational advisors could use to re-orient certain elements of their practices. These helped them 
identify actions they would want to implement more often or differently in their professional practice with regard to 
RBI use. Some educational advisors also reported that the project had helped them discover tools for disseminating 
knowledge and for facilitating groups to encourage discussions about RBI use in the schools. 

School principals. Like the educational advisors, the principals reported that the model was a resource that 
helped them identify actions they could take or improve in their practice with regard to fostering RBI use. However, 
while the project had led some of the principals, as well as some educational advisors, to conclude that they should 
be more proactive in analyzing the needs of teachers in their schools with respect to new knowledge, these 
participants said the project had not provided them with enough concrete resources to conduct this analysis. 

3.1.3 Supporting the Extension of Learning to Other Practitioners in the Participating Schools 

Results showed considerable variation in the extension of learning into the participants’ daily practice. Participants 
also sometimes were constrained by the lack of resources or difficulty in obtaining these. 

Teachers. The actions taken by teachers in their own practice and with colleagues in their schools varied from 
one school to another. In some schools, the teacher and other members of the triad (principal and educational advisor) 
indicated that actions were essentially focused on the teacher’s own practice. 

Educational advisors. The extension of learning also varied greatly among the educational advisors. For 
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example, regarding their role in making knowledge accessible, the results showed that the knowledge disseminated 
depended on the themes targeted and the specific needs of each school. Some educational advisors had to focus more 
on implementing resources for finding, sorting, and adapting knowledge, while others concentrated more on 
knowledge dissemination strategies. That being said, several educational advisors pointed out that the resources had 
been presented to them, but that they had not been able to absorb them sufficiently so that they could apply them in 
action. 

I would need another training session, so that things are less rushed. Because I’m going back to where I 
was already looking for information. All this that they’ve shown us, we’d like someone to come sit with us 
and run through it with us again. It was too fast, but I have the feeling it would help me. -Educational 
advisor (School 1) 

Furthermore, several educational advisors said they had not had enough time and resources during the pilot year to 
be able to support the implementation, monitor it, and evaluate the impacts of RBI use as recommended by the RAC. 

School principals. The principals mainly applied what they had learned with respect to providing time and 
space for teachers to be able to talk about pedagogical matters. As well, many of the principals had encountered 
problems when trying to apply the learning on identifying priority needs for RBI in their schools. In several schools, 
the consultation process was long and tedious, primarily because teachers had difficulty articulating their own 
pedagogical needs. 

In terms of needs, the discussions focused a lot on the material level, on physical spaces, things we don’t 
always have control over, like time or money, whereas there are needs that can be met in other ways. At the 
time, in our school, the discussion wasn’t yet focused very> much on pedagogical issues, so discussions 
about establishing needs weren’t happening at the pedagogical level. -Teacher (School 6) 

Finally, the principals had worked on implementing the recommended practices concerning collaboration and 
networking within their school, but less so on applying what they had learned about collaboration and networking 
outside the school (e.g. with other schools in the school board, parents, and the community). Some principals said in 
the interviews that, while the community of practice had shown them the value of doing this and had provided a first 
contact with peers in other schools who were motivated to create a network, they nevertheless felt that the 
educational advisors were better equipped to interact with the other schools. 

3.2 Which Components of the Action Theory’ Facilitated or Impeded Activation of the Determinants? 

This section identifies, based on content analysis of the interviews, the main elements of the action model that either 
facilitated or impeded activation of the determinants presented in the change model. 

3.2.1 Context 

Analysis of the interviews showed that lack of time, attributable at least in part to budget cutbacks occurring in the 
education sector, affected RAC implementation in participants’ schools. More specifically, several principals said 
they would have liked to be able to free up teachers in their schools more often to foster development of a culture of 
collaboration and RBI integration, but that there were not enough resources or time for this. Several participants also 
reported that lack of time impeded their uptake of the tools and resources provided by the project as well as their 
availability to devote themselves to the project or to collaborate with their triad colleagues to spread what they had 
learned to others in the school. 

3.2.2 Associated Organizations 

In some of the participating schools, the impact of lack of time and resources was lessened by the school board’s 
commitment to the project. In some settings, this commitment allowed the educational advisor to give priority to 
services provided to the participating school. However, this type of commitment by the school board regarding the 
educational advisor’s tasks had not been formally envisioned in the action model, and not all school boards 
undertook this commitment. 

3.2.3 Participant Selection Process 

The school boards’ involvement in the participant selection process was also sometimes an impediment. In essence, 
it appeared that, in certain cases, some of the information transmitted by the scientific committee to the contact 
persons in the school boards about the project, its objectives, and its implementation were only partially passed on to 
the participating schools. As such, several respondents indicated that the criterion regarding voluntary participation 
had not always been fully respected, as some people had been pressured into participating in the project. Several 
participants also reported they have been given very little information about the project’s objectives and process, or 
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about what was expected of them in its implementation. Thus, these participants had to devote some time at the 
beginning of the process to grasp these elements, which affected their initial ability to understand the relevance of the 
project and to plan and structure their approaches, both as individuals and within their triads. 

3.2.4 Intervention Protocol 

One element of the intervention protocol, the community of practice meetings, helped compensate for the initial lack 
of understanding of the project. In fact, several participants reported that the community of practice meetings, and 
more specifically the periods of discussion and sharing among participants that were built in to the start of every 
meeting, helped clarify any grey areas about the objectives and the project implementation process. Participants also 
pointed out other positive influences of these discussions and sharing periods; some said these periods motivated 
them to take action in their school, since they were, to some extent, expected to report back to the group. 

I found we had a bit of everything: I went to get knowledge, I exchanged ideas with people, I brought that 
back to my school and I tried to put it into practice. I find that the balance between these comings and 
goings has been very useful. Much more than if I’d just been given knowledge and came back from that with 
lots of stuff; I’m not sure I would have found that to be so useful. As it was, I found it to be concrete: we 
always brought back experiences from the field. I think the little bit of time we spent at the start of each 
meeting to touch base about what was happening elsewhere was helpful because it forced us to have a 
certain rigour, a certain commitment to that project. -School principal (School 7) 

However, several participants also indicated that these periods at the start of each meeting could have been better 
defined, such as by determining beforehand what topics would be addressed at the next meeting. 

Another weakness in the intervention protocol appeared to be the lack of time devoted in community of practice 
meetings to understanding the RAC, the process, and the accompanying tools. Some participants also considered that 
the one-year duration of the project in itself limited what could be expected in terms of outcomes. They indicated 
that it was not realistic to expect that, in one year, everyone would come to a new understanding of their role in terms 
of RBI use, and of the process and related resources, or that they would have applied all of this in their practice. 

Several respondents also underscored the positive influence of community of practice meetings on participants’ 
perceptions regarding their work in triads (collective competency). In essence, they emphasized the value of 
presenting all the content of these meetings to all the participants, rather than favouring a profession-based approach. 
According to them, this fostered the development of collective competence by helping them develop a shared 
understanding of the project and a deeper understanding of each person’s role and their respective challenges and 
constraints. 

That being said, with respect to the triad’s capacity to apply what they had learned regarding collective competency, 
one element of the intervention protocol—the RAC itself—was perceived as a constraint. In general, participants 
said that the RAC did not provide sufficiently clear and explicit indications about how this collaboration should be 
planned and implemented. Some also pointed to another shortcoming of the RAC, having to do with improving 
teachers’ individual competencies. According to them, it was not clear whether the two competency domains of the 
framework related to teachers applied specifically to the teachers involved in the project as triad members, or to all 
teachers in all the schools. 

I said to myself, these sheets should probably be used by all teachers. The perception I had was that, for the 
teachers who were part of the project, what was written there was pretty much the way things were. There 
were small points where I, personally, said to myself, ‘oh well, that point, I don’t have it’, but most of the 
people who were part of the project were people who wanted to be there, so for them, it was 
obvious. ’’-Teacher (School 4) 

As shown by this participant, there appeared to be a certain amount of confusion in the action model on this point. In 
essence, while the two competency domains described were specifically aimed at the teachers involved in the project, 
the RAC did not appear to offer enough actionable ideas to enable teachers who were already inclined to use RBI, 
like those selected to participate in the project, to improve their practices in that area. This could make it appear that 
the RAC, in terms of these two competency domains, was intended for all teachers rather than for those involved in 
the triads. Yet the action model clearly indicated that the RAC implementation targeted the members of the triads, 
i.e., one school principal, one educational advisor, and one teacher in each participating school. 

3.2.5 Target Population 

Participants’ views were mixed regarding the decision to target a triad in each school rather than to disseminate the 
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RAC documents to all teachers and personnel in the schools. Several thought it could have been useful to 
disseminate these documents to all teachers and personnel, and that this would have helped the other teachers on the 
school teams to understand what was expected of them and the resources available to them for using RBI as part of 
their professional development. Some participants also suggested that wider dissemination would have helped frame 
the intervention being conducted by the teachers in the project for the other teachers in their school. That being said, 
it seemed that such wider dissemination would have been conceivable only on three conditions: 1) if the school team 
were already positively inclined toward RBI use; 2) if the teachers and other school personnel were given an 
orientation so that they could understand the value and spirit of the RAC; and finally, 3) if the RAC could be made 
available in an abbreviated format. 

3.2.6 Implementers 

Several participants reported that the project leaders’ ability to listen, as well as their sincere desire to supply them 
with what they needed and to support them in implementing initiatives in their school was a major motivating factor 
in the project. According to them, the leaders were approachable, welcoming, and available. Several participants also 
noted that the leaders had a good mix of complementary expertise, which enriched both the content and the group 
facilitation over the implementation year. Finally, several participants explained that these people truly sought to put 
participants’ own experiential knowledge to good use, and in that way, to help them mesh their experiential 
knowledge of RBI use with new knowledge acquired during the project. However, some respondents noted that 
putting their experiential knowledge to use sometimes came at the cost of delaying the development of tools 
provided as part of the project. These delays were due to the project leaders’ desire to take into account participants’ 
needs and experiential knowledge in developing the process and tools. To achieve this, project leaders took the time 
to consult regularly with triad members, but these consultations and the resulting adjustments led to delays in 
transmitting the final versions of documents to the participating triads. These delays may have affected deployment 
of the change model. 

4. Discussion 

The objective of this study was to respond to two evaluative questions: 1) What were the proximal outcomes of the 
RAC implementation, i.e., what determinants were activated by this implementation? 2) Which components of the 
action theory facilitated or impeded activation of the determinants? 

In response to the first question, the results showed that the first determinant, influencing participants’ perceptions 
regarding their role in the process of fostering RBI use, was activated, but differently for the various professionals 
targeted (teachers, educational advisors, principals). The other two determinants, increasing participants’ resources 
with regard to RBI use and supporting the extension of learning to other practitioners in the participating schools, 
were also activated, but only partially. Both with respect to increasing resources and to extending the knowledge, 
considerable variation was observed among participants. 

In response to the second question, the results indicated that the activation of determinants and achievement of 
objectives were probably impeded by the lack of time and resources in the education sector, and by certain 
components of the intervention protocol, such as insufficient activities to help participants take ownership of the 
process and tools, as well as certain gaps in the RAC and misunderstandings regarding the participant selection 
process. On the other hand, the support provided by some school boards, as well as certain strengths of the 
intervention protocol, such as the RAC’s comprehensiveness and flexibility, the discussion periods in community of 
practice meetings, and the open attitudes of the project leaders, appeared to have been helpful in activating the 
determinants and achieving the objective of strengthening participants’ individual and collective competencies. 

With respect to both the activation of the determinants of change (first question) and how the action model 
components influenced this activation (second question), considerable variation was observed from one participant to 
another, as well as from one school to another. These variations were consistent with Durlak’s (2015) observation 
that there are often significant gaps between how a program is conceptualized and how it is operationalized on the 
ground. That being said, it is important to pay attention to these variations, as many authors have noted they can have 
consequences for the potential outcomes of an intervention (Aarons, Sommerfeld, Hecht, Silovsky, & Chaffin, 2009; 
DuBois, Holloway, Valentine, & Cooper, 2002; Durlak & DuPre, 2008; Smith, Scheider, Smith, & Ananiadou, 2004; 
Tobler, 1986; Wilson, Lipsey, & Derzon, 2003). 

To analyze the sources of such variability and to minimize them whenever possible, Meyers, Durlak, and 
Wandersman (2012) propose a model, the Quality Implementation Framework (QIF). This model identifies 14 steps, 
grouped into four phases, to support high-quality implementation of a project. This model is particularly useful to 
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structure reflection on the results of the present study and to draw conclusions regarding the intervention and future 
studies, for three reasons: 1) it is based on a review of models that already exist in several domains (education, health, 
management); 2) it reveals the convergence among several of these models; and 3) it focuses on the ‘how to’ in order 
to foster better implementation that will advance not only science, but also practice in this area (Meyers et al., 2012). 

The first of the four phases presented by the QIF has to do with initial considerations regarding the host setting. This 
phase includes eight of the 14 steps presented in the model. This is a major phase that must be completed before 
implementing the intervention. Its steps address the questions: To what extent does the target public understand what 
is involved in implementing the intervention? Do the persons being targeted have an appropriate and realistic 
understanding of the potential outcomes of the intervention? Are the targeted persons and their organization ready to 
support the intervention adequately? To respond to these questions, the suggested steps involve, among other things, 
assessing the needs and available resources, determining whether the intervention needs to be adapted in any way to 
ensure it meets the needs of the target setting, and obtaining a formal commitment from stakeholders. Some of the 
present study’s results are best understood when considered in light of this phase. In essence, the results indicated 
that the triad members could have benefited from having a better understanding of the intervention (its objectives and 
processes) at the outset. Moreover, it might be useful to rethink the potential components and inalterable actions of 
the intervention: do some aspects of the acquired knowledge absolutely have to be disseminated by the triads through 
their actions with their school teams, if they are to have an impact on competency levels and potentially on RBI use 
in the schools involved? According to Meyers and his colleagues (2012), this reflection should be carried out by the 
intervention leaders to guide any adaptations and variations in the implementation. 

The second phase of the QIF is creating a structure for implementation. This phase involves two steps: creating an 
implementation team and developing an implementation plan. This phase addresses the questions: 1) Is there a clear 
plan regarding what should happen and when? 2) Who will perform the different tasks related to the implementation, 
and who will monitor and supervise this implementation? These steps of the QIF are consistent with suggestions 
made by participants in the present study to the effect that spreading their knowledge might have been helped by 
better planning on the part of the triads involved. 

The third phase of the QIF is having an ongoing structure once implementation begins. This phase consists of three 
steps: providing ongoing assistance to participants in implementing the intervention, continuously monitoring that 
intervention, and setting up feedback mechanisms. With regard to this phase, the results of the present study showed 
that the action model envisioned ongoing assistance to the participating triads in three forms: discussions in 
community of practice meetings, tracking sheets for participants, and individual support offered to participants who 
expressed the need to one or another of the project leaders. While the community of practice discussions and 
individual support appeared to have been helpful, the tracking sheets were less successful. Use of this tool was 
clearly seen to vary from one school to another, and it was generally the educational advisors who used it. By that 
very fact, its utility was diminished. 

Finally, the fourth phase of the QIF involves improving future applications. This phase consists of a single step: 
learning from the experience. It is intended to respond to the question: What can we learn about the quality of the 
implementation following this project? It was in the context of this phase that the present study was conceived and 
conducted. The study provided a deeper understanding of the strengths and limitations of the RAC implementation 
pilot project and established to what extent this intervention was able to activate, among participants, the 
determinants of change to improve their individual and collective competencies for using RBI in their schools. It also 
identified possible improvements to the intervention in accordance with the three phases presented above that could, 
in future iterations of the project, enhance the stability of the RAC implementation and its outcomes. 

There are four possible improvements: 

1) Before the start of the implementation, ensure participants clearly understand the objectives and procedures of the 
intervention, by improving communications during the participant selection process and by determining what are the 
inalterable components of the intervention. 

2) Provide tools and support to the participating triads in planning how they will extend their actions into their 
schools. 

3) Strengthen the mechanisms for monitoring and providing feedback on the intervention. 

4) Continue to evaluate and learn about the RAC implementation and its outcomes and about competency models in 
general. Suggestions in this regard are offered in the section on ‘Research avenues and future interventions’ in the 
present article. 
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4.1 Strengths and Weaknesses of This Study 

This study was based on the theory-driven integrative approach to process and outcomes evaluation, as described by 
Chen (2015). That author recommends performing this type of evaluation using a mixed method in which qualitative 
data are used to clarify program theory and to highlight the factors that facilitate or impede activation of 
determinants and achievement of objectives, while quantitative data are used to confirm whether or not the 
determinants were activated and the objectives achieved. However, given that the present study was, to the authors’ 
knowledge, a first on the subject of evaluating the processes and outcomes of implementing a competency model to 
foster RBI use in education, it appeared appropriate to use a more exploratory qualitative methodology, so that 
participants would not be constrained in expressing what they considered to be the processes and outcomes of the 
RAC implementation. That being said, while the present study was limited to a qualitative methodology, it was 
conducted using a variety of tools for data collection and analysis to ensure data triangulation. 

The theory-driven integrative approach to process and outcomes evaluation (Chen, 2015) also suggests evaluating all 
of the links present in the program theory. The present study, however, examined only a single link, between the 
action model and the activation of determinants, by focusing on two questions: 1) Did the action plan 
implementation help activate the determinants? 2) Which components of the action model facilitated or impeded that 
activation? As such, the present study did not identify the effects of activating the determinants on the subsequent 
components of the program theory. Because of this, several questions remain, including: 1) Did activating the 
determinants actually improve the individual and collective competencies of triad members with regard to RBI use? 
2) Did improving competencies in the triad members lead to an increase in RBI use in those triads’ schools? 

4.2 Research Avenues and Future Interventions 

Given the points discussed and the limitations described in the preceding sections, it appears that the conclusions of 
this study are preliminary and that other studies are required on the processes and outcomes of implementing the 
RAC. 

In particular, they indicate that it would be useful, in the future, to examine in greater depth the causes and effects of 
the variations seen in the activation of determinants. This could be useful in better targeting any adaptations needed 
to the RAC implementation according to types of professionals or to each school’s specific context. It would also 
provide a better understanding of which elements of the RAC and its implementation are imperative for activating 
the determinants and for achieving desired outcomes. 

By identifying with greater certainty potential adaptations to the RAC and to its implementation according to specific 
school contexts and selected professionals, and by identifying clearly which elements are core to the intervention and 
cannot be changed, leaders of a future iteration of this intervention will be better able to support the persons and 
schools involved in its implementation. 

Further studies should also be conducted to determine whether activating the determinants of change during the RAC 
implementation is successful in terms of improving not only the competencies of triad members, but also RBI use in 
their schools. 

5. Conclusion 

According to Chen (2015), an integrative evaluation of processes and outcomes is able to shed light on the outcomes 
of an intervention and how they are produced, so that stakeholders can be better informed in making decisions 
regarding that intervention. While the information provided here is only partial, as this study only looked at how the 
intervention’s implementation influenced activation of determinants of change and did not focus on more distal 
outcomes, it is nevertheless relevant. The results of the present study suggest recommendations for the next 
implementation of the intervention. In fact, the CTREQ and the RENARD team have already applied several of these 
recommendations in producing a final version of the Competency Model for Knowledge Translation to Support 
Educational Achievement among Quebec Youth (RAC) and accompanying tools. The results of the present study 
also contribute to the advancement of knowledge on the implementation of competency models, given that, to date, 
little knowledge is available on how targeted individuals perceive competency frameworks and what outcomes they 
attribute to the implementation of such frameworks (Caldwell, 2008; Heinsman et al., 2008; Redmond, 2013). In this 
respect, the present study is a first step in better understanding the proximal outcomes of implementing a competency 
model and the processes influencing them. 
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Note 

Note 1. The evaluation methods had been presented in community of practice meetings and participants were advised 
that they would receive an invitation to participate. 
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